s3: fix S3 Object Lock header issue for lock file writes #36120
Merged
+343
−2
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When S3 Object Lock is enabled on a bucket with a retention period, S3 requires the Content-MD5 or x-amz-sdk-checksum-algorithm header for object uploads (via PutObject) to ensure data integrity during the upload process. Terraform’s state writes to the S3 bucket relied on the “uploader” from aws-sdk-go-v2, which automatically appends these required headers. However, the lock file implementation did not use the “uploader,” resulting in missing headers for PutObject requests and conflicts with Object Lock requirements. This commit updates the lock file implementation to use the “uploader,” ensuring the necessary headers are included in the requests, maintaining compatibility with Object Lock-enabled buckets.
…raform into b/s3-object-lock-file
bschaatsbergen
added
the
1.10-backport
bschaatsbergen
force-pushed
the
b/s3-object-lock-file
branch
from
e43bf6e to
9db9647
Compare
bschaatsbergen
changed the title
jar-b
approved these changes
Dec 2, 2024
|
This will happen on the backport PR. |
|
Reminder for the merging maintainer: if this is a user-visible change, please update the changelog on the appropriate release branch. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #36113
When S3 Object Lock is enabled on a bucket with a retention period, Amazon S3 requires the
Content-MD5orx-amz-sdk-checksum-algorithmheader to be present in object uploads (PutObject). See Uploading objects to an Object Lock enabled bucket.It seems we overlooked maintaining the default behavior of the
skip_checksumflag for the lock file when writing to S3 Object Lock-enabled buckets.To clarify the default behavior of
skip_checksum: by default, if this argument is not set in the backend, we set the S3 checksum algorithm behavior toSHA256. This causes the underlying S3 AWS SDK V2 serializers to automatically append that requiredx-amz-sdk-checksum-algorithmheader. For more details, see the relevant code in the AWS SDK v2 serializers.This PR updates the lock file implementation to use the same "uploader" that we rely on for writing Terraform state to S3, and preserving the default
skip_checksumbehavior for the lock file. To ensure a consistent and compatible experience with S3 Object Lock-enabled buckets between the two mechanisms writing data to S3.